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CONVERSION  OF  IRAMMP  DATA  TO  HDF  FORMAT 


I.  Introduction 

In  the  last  few  years  a  public-domain  standard  format  for 
the  storage  of  scientific  data,  images,  and  auxiliary  information 
has  gained  popular  acceptance  [1].  This  format,  developed  at 
NCSA  -  University  of  Illinois,  is  called  Hierarchical  Data  Format 
(HDF) .  The  HDF  format  is  an  attractive  alternative  to  the  NATO 
format  for  the  storage  of  IRAMMP  data  for  two  reasons.  First, 

HDF  allows  a  convenient  means  for  storage  of  auxiliary  data  in 
the  same  file  as  the  basic  image  data.  In  the  case  of  IRAMMP 
this  includes  the  time  stamp  information  in  addition  to  general 
information  such  as  filter  setting,  date,  laboratory,  etc. 

Second,  a  variety  of  powerful  public-domain  and  commercial  data 
visualization  software  packages  can  read  and  manipulate  HDF 
files.  The  public-domain  packages  include  the  NCSA  developed 
Xlmage  and  XDS.  A  commercial  firm  has  developed  value-added 
versions  of  these  programs  for  a  variety  of  popular  hardware 
platforms  including  Macintosh  and  UNIX  workstations.  Modules 
allowing  most  data  visualization  software  packages  to  read  HDF 
files  exist  or  will  soon  be  available.  The  HDF  standard  is 
currently  under  consideration  by  the  National  Science  Foundation 
(NSF)  for  possible  merger  with  the  netCDF  data  format.  The 
netCDF  format  is  itself  based  on  a  NASA  format  called  CDF.  This 
development  makes  it  likely  that  HDF  format  files  will  become 
accessible  to  an  increasingly  wide  range  of  application  software 
packages . 

It  is  the  purpose  of  this  report  to  describe  the 
implementation  of  HDF  for  IRAMMP  data  developed  at  the  Naval 
Research  Laboratory.  The  report  also  provides  information  on  how 
to  acquire  the  public-domain  programs  Xlmage  and  XDS  from  NCSA 
via  the  Internet. 

II.  IRAMMP  Data  -  existing  NATO  format 

The  current  distribution  format  for  IRAMMP  data  uses  the 
NATO  format.  This  is  an  older  standard  format  developed  to  allow 
interchange  of  data  among  the  various  NATO  technical  working 
groups.  The  chief  virtue  of  the  NATO  format  is  that  it  is  a 
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machine  independent,  nine  track  tape  compatible,  standard  data 
format.  It  is  not  ideally  suited  to  the  newer  computing 
environment  characterized  by  workstations  running  sophisticated 
graphical  user  interfaces  and  data  visualization  software 
packages.  To  provide  background  for  the  HDF  implementation,  this 
section  reviews  the  manner  in  which  IRAMMP  data  is  stored  using 
the  NATO  format. 

The  IRAMMP  sensor  is  a  120  detector  scanner.  Images  are 
thus  120  rows  by  M  columns.  Each  colvunn  represents  a  time 
sample.  The  sensor  is  usually  operated  in  such  a  manner  as  to 
produce  M  =  1455  samples  per  image.  Actually,  the  IRAMMP  sensor 
simultaneously  records  both  longwave  IR  (LWIR)  and  midwave  IR 
(MWIR)  images.  However,  these  two  images  are  recorded 
separately.  Consequently  this  detail  is  not  germane  to  the 
subject  of  this  report.  The  image  specific  value  of  M  is 
recorded  in  a  header  to  be  described  below.  The  NATO  format 
files  that  are  distributed  to  the  general  user  community  contain 
calibrated  data.  The  units  associated  with  each  data  value  are 
banded  radiance  (watt/sqm/sr) . 

The  NATO  format  IRAMMP  files  are  organized  as  a  sequence  of 
256  byte  records.  The  images  are  stored  sequentially  with  each 
image  comprising  the  same  record  structure.  This  structure  is: 

1)  a  single  record  standard  format  NATO  header  containing  such 
information  as  the  date  on  which  the  data  was  recorded  and  the 
integer  M,  2)  a  two  record  IRAMMP  header  containing  the  power 
integer,  P,  to  be  described  below,  and  other  auxiliary  data,  3) 
one  or  more  blocks  of  64  records  (23  blocks  typical)  containing 
the  data  proper.  The  exact  number  of  blocks  depends  on  M.  The 
extra  room  in  the  last  block  is  filled  with  zeros.  Most  of  the 
information  in  the  headers  is  not  useful  for  analytic  work. 

The  three  header  records  are  encoded  as  ASCII  character 
information.  The  encoding  of  the  data  records  is  more  complex. 
Each  2»C  byte  record  represents  a  single  time  sample  of  IRAMMP 
data.  The  first  16  byLes  are  a  preamble  containing  a  packed  BCD 
representation  of  the  IRIG  time  code  associated  with  that  sample 
along  with  BCD  codes  for  sensor  head  temperature  readings  and 
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other  similar  information.  In  the  remaining  240  bytes  are  stored 
120  sixteen  bit  integers  representing  the  radiance  values 
associated  with  the  120  detectors  of  the  IRAMMP  sensor.  The 
radiance  (watt/sqm/sr)  associated  with  each  integer  is  the 
integer  value  multiplied  by  lo'**  ,  where  P  the  power  integer 
encoded  as  an  ASCII  character (s)  in  the  IRAMMP  header.  P  is 
chosen  in  such  as  way  as  to  insure  that  no  integer  exceeds  32767. 
Typically  all  the  integers  lie  in  the  range  16384  to  32767. 

For  detailed  information  on  this  format,  including  the 
structure  of  the  header  records,  see  ref  2. 

The  main  drawbacks  of  this  format  from  the  standpoint  of 
data  visualization  software  can  be  seen  from  this  brief  review. 
They  are:  1)  The  image  size,  M,  and  the  power  integer,  P,  are 
buried  in  unconventionally  formatted  header  records,  2)  The  data 
records  have  very  non-standard  structure  due  to  the  16  byte 
preamble,  3)  The  IRIG  time  codes  are  comparatively  inaccessible 
because  of  a  non  standard  encoding  using  packed  BCD. 

III.  HDF  Implementation 

The  HDF  standard  is  quite  flexible.  As  is  usually  the  case 
with  a  flexible  standard,  specific  choices  must  be  made  to 
realize  an  implementation.  Details  of  the  implementation 
developed  for  the  IRAMMP  data  are  given  in  the  Appendix.  This 
section  gives  an  overview  of  the  design. 

In  the  case  of  IRAMMP  data  the  most  important  decision 
concerned  whether  to  encode  the  data  as  integer  or  floating  point 
numbers.  In  order  to  retain  the  scale  factor  information, 
represented  by  P  in  the  NATO  format,  it  was  decided  to  use 
floating  point  numbers  which  directly  represent  the  radiances  in 
watt/sqm/sr.  In  HDF  this  is  conveniently  done  using  a  structure 
called  a  scientific  data  set  (SDS) .  This  choice  is  less 
efficient  from  a  storage  point  of  view  than  the  integer 
alternative,  called  a  Raster  image  set.  Testing,  albeit  limited, 
has  shown  that  standard  UNIX  file  compression  utilities  recover 
most  of  the  factor  of  two  inefficiency.  It  was  further  decided 
that  each  image  would  be  stored  in  a  separate  SDS.  This  choice 
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was  not  forced  because  an  SDS  can  store  either  two  or  three 
dimensional  data.  However,  an  SDS  containing  the  typical  group 
of  5  to  10  consecutive  IRAMMP  images  would  be  quite  large  and 
unwieldy  for  all  but  the  most  powerful  workstation  platforms. 

It  was  decided  to  unpack  the  time  codes  and  store  them  as  an 
HDF  data  structure  called  a  data  identifier  annotation.  This  is 
also  not  optimal  from  the  storage  point  of  view,  but  here  the 
increased  storage  requirement  is  minimal.  The  useful  information 
from  the  NATO  format  headers  is  stored  in  HDF  an  file  descriptor 
structure . 

IV.  NATO  to  HDF  conversion  program 

A  C  language  program  called  nato2hdf  has  been  written  to 
convert  NATO  format  files  to  HDF  files.  This  program  is  designed 
to  run  on  a  workstation  running  UNIX.  The  calling  sequence  is 
nato2hdf  [options]  <inputfile>.  The  options  are:  -o  <outfile 
name>;  -p  <palfile>;  -s  <start  image>;  -e  <end  image>.  The 
input file  is  the  input  file  containing  the  NATO  format  data.  The 
outfile  is  the  output  HDF  formatted  file.  The  default  output 
file  name  is  nato.hdf.  The  s  and  e  options  are  included  because 
a  NATO  format  file  can  have  any  nvimber  of  IRAMMP  images.  The 
user  of  nato2hdf  can  select  any  contiguous  range  of  these  to  put 
into  the  HDF  file.  The  default  is  to  put  all  the  images  in  the 
NATO  format  file  into  the  HDF  file  as  one  image  per  SDS.  The  p 
option  allows  a  color  pallet  to  be  stored  with  the  data  as  part 
of  the  HDF  file.  Programs  such  as  Xlmage  can  store  other  pallets 
in  the  HDF  file  once  it  is  created.  Once  started,  the  program 
run  autonomously  reporting  on  progress  as  it  proceeds.  The 
nato2hdf  program  is  available  on  request. 

V.  Public-Domain  Software 

The  HDF  utility  library  and  data  visualization  programs 
Xlmage  and  XDS  are  available  from  NCSA  which  is  located  at  the 
University  of  Illinois,  Urbana-Charopaign.  To  use  HDF  software, 
access  to  one  of  the  following  is  needed:  a  Sun  System  3  or  4 
running  UNIX,  a  Silicon  Graphics  system  running  UNIX,  a  VAX 
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computer  running  UNIX  or  VMS,  a  Macintosh  running  MacOS,  or  a  IBM 
compatible  PC  running  DOS.  Other  workstations  running  UNIX  are 
also  useable  with  some  extra  effort.  At  least  8  MByte  of  memory 
is  recommended  together  with  a  comparable  amount  of  disk  space. 

To  obtain  HDF  software  over  the  network,  access  to  a 
computer  linked  to  the  Internet  and  running  the  ftp  file  transfer 
program  is  required.  The  procedure  is  to  open  an  ftp  connection 
to  ftp.ncsa.uiuc.edu.  On  a  UNIX  platform  this  is  done  by  issuing 
the  command  ftp  ftp.ncsa.uiuc.edu.  The  next  step  is  to  login 
using  the  login  name  of  anonymous  and  any  identifying  name  as  the 
password.  README  files  on  the  ftp.ncsa.uiuc.edu  machine  explain 
how  to  obtain  the  required  files  using  the  ftp  get  command. 

The  HDF  utility  library  is  required  by  those  users  wishing 
to  write  their  own  programs  to  manipulate  the  data  in  HDF  files. 
In  the  case  of  IRAMMP  HDF  files,  such  a  user  written  program  is 
required  to  get  access  to  the  time  stamp  data  stored  in  the  HDF 
file.  The  HDF  library  is  constructed  such  that  these  user 
written  programs  can  use  either  FORTRAN  or  C  as  the  programming 
language. 

Both  the  Xlmage  and  XDS  programs  can  display  the  IRAMMP  HDF 
files  directly.  To  use  these  programs  a  workstation  running  some 
version  of  the  X  windowing  protocol  is  required.  Since  an  IRAMMP 
SDS  is  a  two  dimensional  image,  the  Xlmage  program  is  preferred. 
In  addition  to  displaying  the  image,  Xlmage  supports  a  variety  of 
image  processing  functions  such  as  histogram  and  profile  plot 
generation.  No  serious  problems  with  Xlmage  have  been 
encountered  in  testing  with  IRAMMP  HDF  images.  The  XDS  program 
can  be  made  to  work,  however  undocumented  extra  steps  are 
required  to  get  it  to  function  using  Sun's  OpenWindows  version  of 
the  X  protocol.  The  Spyglass  corporation  has  developed  a  value 
added  version  of  Xlmage  called  Transform.  Transform  is  available 
for  the  Macintosh  as  well  as  most  UNIX  platforms. 

APPENDIX 

This  appendix  covers  the  specific  implementation  of  HDF  used 
by  the  nato2hdf  program.  It  is  intended  for  experienced  users 
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and  should  be  read  in  concert  with  ref  l.  The  most  concise  way 
to  describe  the  contents  of  a  HDF  file  is  to  list  the  types  of 
HDF  tags  that  it  contains  and  the  data  items  to  which  the  tags 
point.  The  following  table  1  lists  the  tags  created  by  nato2hdf 
and  the  data  to  which  they  point 

Table  1 


Tag 

type 

IRAMMP  data 

FID 

file  identifier 

IRAMMP  scene  number 

FD 

file  descriptor 

bytes  1-48  of  IRAMMP  header  +  date 

IPS 

Image  Palette 

optional  user  supplied  palette 

SDS 

scientific  data 

set  IRAMMP  image  (floating  point) 

DIA 

data  annotation 

IRAMMP  time  stamps 

There  is  only  one  FID  and  FD  tag  per  file.  There  are  as  many 
SDS  and  DIA  tags  as  there  are  images  in  the  input  NATO  file.  The 
palette  is  present  only  if  the  -p  option  is  selected.  There  are 
additional  tags  associated  with  each  SDS.  These  are  the 
dimensions  of  the  SDS,  the  labels  and  the  units.  The  dimensions 
are  120  and  M.  The  labels  are  ”column”,  "row”  and  "radiance”. 

The  units  are  "sample",  "detector"  and  "watt/sqm/sr" . 
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